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Whole slide image (WSI) processing is a common technique used in the 
analysis process performed by pathologists. Identifying precise and accurate 
regions of cancerous in the tissue is an important process in the disease 
diagnosis modality. This work proposes an automated technique for 
identifying invasive ductal carcinoma (IDC) in histology images using. An 


image is divided into small non-overlapped patches (or image windows). 
Then, the task is to classify the image patches into different classes, i.e., i) 
IDC and ii) non-IDC. We employ a two-stage classification-based to classify 
the patches, as to identify IDC regions in the tissue. In the first stage (patch- 
level classification), image patch classification is carried out using a 
conventional handcrafted feature and deep-learning technique are explored. 
The second stage (post-processing) undergoes a refinement process, which 
considers the spatial relationships between the neighboring patches. This 
stage aims to amend some of miss-classified patches. Markov random field 
(MRF) is implemented in this stage to examine the relationships of the 
patches and their neighborhoods. The experiments are conducted on public 
dataset. The experimental results show the post-processing can improve the 
performance of the classification in the first stage using the handcrafted- 
based technique and deep learning. 


Keywords: 


Breast cancer classification 
Computer aided diagnosis 
Deep neural network 
Histopathology image analysis 
Morphological features 


This is an open access article under the CC BY-SA license. 


Corresponding Author: 


Phatthanaphong Chomphuwiset 

Department of Computer Science, Faculty of Informatics, Mahasarakham University 
Maha Sarakham, Thailand 

Email: phatthanaphong.c @msu.ac.th 


1. INTRODUCTION 

Histology image analysis is a clinical procedure that is performed to study and understand the 
histology of tissue samples. In the conventional process, tissue samples are collected from body organs. 
Then, a sequence of the preparation process is carried out (i.e., fixation, dehydration, clearing, infiltration, 
embedding, sectioning, and staining) [1]-[4]. After the preparation process, they are mounted to glass slides 
for the examination under a light microscope. A whole slide analysis is a common technique that has been 
applied during examination process (by pathologists). The examination is usually performed with various 
magnifications (different scales) to obtain an essential meaningful interpretation of tissue histology. The 
invention of digital tissue images has immensely contributed to the modern pathology that allows 
pathologists to be able to perform their tasks in examining and analyzing the digital tissue slides in many 
diagnostic applications [5]. As a result, the analysis for disease prognosis and progression can be carried out 
more expediently and productively. This includes, for example, disease prognosis, severity and malignancy 
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level evaluation, and tissue stating and grading. Associating with clinical data of patients, further proper 
treatment can be, as a consequence, planned efficiently. 

In modern histology image analysis, an automated process of screening systems assists pathologists 
in the diagnosis modality. Invasive ductal carcinoma (IDC) is the most common subtype of all breast cancer 
(BCa) is an example [6]. Identifying IDC in the tissue images can be substantial to help in the diagnosis 
process and evaluate the severity of cancer. In the WSI examination, the resolution of the image imposes the 
handwork that is required in the analysis process. With the emergence of AI and technology, an automated 
system for identifying IDC in the tissue images can be implemented [7]-[11]. In general, the identification 
can be carried out using a classification-based technique. The image of tissues is divided into non-overlapped 
patches. Each of the patches is then fed to a classifier to identify whether it is IDC or vice versa. Therefore, 
this work proposes a two-stage classification-based to classify the patches, and yet to identify IDC regions 
the tissue. In the first stage, image patch classification is carried out using a conventional handcrafted feature 
(based around on texture descriptors) and deep-learning techniques are explored. The second stage undergoes 
a refinement process, which considers the spatial relationships between the neighboring patches. This stage 
aims to adjust the miss-classified patches to correct classes. Markov random field (MRF) is implemented in 
this stage, taking into account the relationships of the neighboring patches [12], [13]. 

The rest of this paper is organized as follows: section 2 provides related work and the literature in 
histological image analysis techniques. Section 3 explains the proposed technique of classifying the patches 
using 1) conventional feature extraction and learning algorithms, ii) knowledge transferring approaches using 
deep learning-based methods, and iii) in-cooperating spatial information of the patches in the classification. 
Section 4 provides the experimental results and the evaluation of the proposed method before the conclusion 
is given in section 5. 


2. RELATED WORKS 

Artificial intelligence (AI) pathology for digital pathological image analysis is usually developed by 
applying image processing and machine learning techniques to implement computer aided diagnosis (CAD) 
applications aiming at assisting pathologists to perform their tasks [1]. These applications are usually 
decomposed by different level-of processing, ranging from low-level processing (e.g., detecting and 
segmentation [14], [15]) and high-level analysis (e.g., disease stating, classification and quantitative 
assessment of tissues [6]-[11], [16]-[22]). 

Detecting and segmenting cellular structures in the images is one of low-level processes that is 
usually required for further high-level analysis, e.g., extracting structure semantics for the analysis. Color and 
morphology-based approaches have been reported in the literature as the pioneer techniques applied to detect 
or segment the structures in histology images [5], [23]. These techniques are not able to produce promising 
results as they cannot cope with the variation of histology images resulted from the preparing process, for 
instance, uneven staining. Watershed and Level-Set technique have also been one of the alternative methods 
applied in the low-level state for detecting some important cellular structures in the images [5]. This 
computational method usually confronts a number of problems, e.g., seed initialization and touching cellular 
structures; therefore, the technique is not able to produce promising results and computational burden [23]. 
As deep learning is increasingly becoming method that have been implemented in various computer vision 
application, including histological image analysis [15]-[16], [23]-[25]. The technique, e.g., U-Net, has 
applied to detect different types of cellular, e.g., cell nuclei, blood vessels, epithelium, and lymphocytes [24]. 
The deep learning approaches have reported for producing promising outcomes in detecting some cellular 
structures in the images. 

Tissue classification is one of the common tasks in histology image analysis that has been in the 
attention of the community. The classification can generally be used in the detection process [10], [11], [17]- 
[22]. Textural features are one of the techniques that have been applied in the classification process [1]. 
These handcrafted feature extractors are performed to represent local/global appearance of the image before 
they are fed to leaning algorithms [1]. In recent years, ensemble-based techniques have also been introduced 
to improve the performance of a single classifier. With the variation of the images (color, light, scale, and 
conventional variation), the handcrafted-based technique can produce promising results in some dataset [1]. 
Deep learning (DL) have been applied to the tasks in histology image analysis, including classification 
process [10], [17]-[20]. 


VGGI16 (CNN architecture) have been used in classifying sub-image of histopathological, image 
into different types, such as IDC and non-IDC [9]. The work divides the whole image into small non- 
overlapped sub-images. Each of the patches is fed to the trained model (by VGGI16). They report some 
promising results in the classification on a public dataset. N. Brancati et.al. propose a technique for Breast 
Invasive Ductal Carcinoma Detection and Lymphoma using a deep learning technique [10]. They employ 
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FusionNet (a residual convolutional neural network) to performance the classification in the image patches 
that have been partitioned from the whole slides. The classification is carried out to classify the image 
patches into IDC and non-IDC, associate with three classes of lymphoma sub-types. 

The classification on sub-images of the whole-slide histology images can be improved by 
determining the relationships of a sub-image (image patch) with its neighboring patches. The type of the 
patch (in a certain scale) tends to be marginally the same with the patches around it. Therefore, this paper is 
aimed at constructing a post-processing method that examines the relationships of the patches for improving 
the classification performance. 


3. DATA AND METHOD 

Invasive ductal carcinoma (IDC) is a common subtypes of all breast cancers [6]. In practice, the 
evaluation the aggressiveness grade of the sample is performed by the specialist pathologists that examine 
mounted samples on the regions typically contain with the IDC. Therefore, one of the common tasks for 
automatedly determine the aggressiveness grades of the tissue is to delineate the exact regions of IDC inside 
of a whole mount slide. This section describes the technique for classifying the image patches. 


3.1. Data description 

Classifying image patches in histology images of breast histology. A histology image of breast 
tissue is separated into small image patches (image widows) with equal size. Each of image patch (p) will be 
classified into two types, i.e., invasive ductal carcinoma (IDC) or negative patch and non-invasive ductal 
carcinoma (non-IDC) or positive patch. An example is depicted in Figures 1(a) and 1(b). 


fas Negative Patches 


a Positive Patches 


Figure 1. Example of patches in a whole slide image of breast histology. The patches are classified into 2 
classes (by the experts), i.e., (a) IDC and (b) Non-IDC [3] 


Thid work uses a public dataset [9], [10] comprising 72,120 image of positive and negative IDC. 
The image patches are collected from several slides undergoing breast cancer at x5 of resolution. Background 
patches are eradicated from the dataset (see Figure 1). 


3.2. Patch-level classification 

Classifying the image patches is straightforward. In general scheme of classification methods, a set 
of objective features (p(x) € RÎ) is firstly extracted from a patch. The classification can be formed by 
constructing a learning function (f: (t) > C) that maps the input features (p(t) to a patch class (C). The 
mapping function can be implemented in many forms. In this work, two techniques of classification are 
applied, i.e., conventional machine-learning method and deep learning-based method [Joseph]. Conventional 
machine-learning method. 

Handcrafted feature for classification textural features, i.e., Grayscale level cooccurrence matrix 
(GLCM) and HOG [26], [27] are implemented in this work. An image patch is converted to a grayscale 
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image before the texture-features are extracted, @(x). The classification on the extracted features is 
performed using a Random Forest classifier that classifies that patch into 2 types, f: P(t) > C. 

Deep leaning classification in general, CNNs have demonstrated a success and drawn to the 
classification problem. The techniques model a learning function that computationally maps an input image 
t € R’*wx4 (where h and w is the image resolution and d is the image depth) in output probability vectors 
y E€ RS, C is the number of classes, in this work C = 2 (i.e., IDC and non-IDC). Each layer, (1), of the 
architecture derives a transformation module, which is associated with a set of adjustable parameters 
followed by non-linear operation, call activation process: 


tı = f,(W, * t_1 + by), (1) 


where 1 <1 < L denotes the layer index and L is the number of layers in the networks, tg is an input image, 
W, and b, are model parameters at layer l, and * denotes a convolution operator with pre-defined kernels for 
convolution layers. fı is a layer specific non-linearity (i.e. defined by activation functions), composed of 


Rectify Linear Unit (ReLU), which is defined as (2): 
ReLU(t,)= (max(0, tı), (2) 
and max-pooling mechanism-a local response normalization-or drop-out. The output of the last layer, pz, is 


input to a softmax function, which outputs a probability vector over the target classes. The classification can 
be performed by minimizing a loss function with respect the network weights (w) as (3). 


argmin L((p), W) (3) 


The loss function (L) is evaluated by determining the difference of the prediction obtained by the 
network (W) and the ground truth data or targets. This loss function can be implemented using different 
techniques, for instance, square loss, logistic loss, exponential loss, and hinge loss. To generalize the 
network, an optimization technique is applied through training processes. Examples of the optimization 
techniques are Stochastic Gradient Descent, RMSProp and Adam [18]. DiagramVGG16 (CNN architecture) 


as shown in Figure 2. 


| | Convolution Layer TE Fully Connected Layer Fully Connected 
m Max Pooling Layer Soft Max Layer 


Figure 2. Diagram of VGG16 architecture [28] 
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3.3. Integrating spatial location of patches 

After the classifiation is peformed, this work impleement Markov random field to refine the 
classification results. The technique examines the relationships of the patch images and its neighbors [12]. 
For a given set of the patch image in an image, there exists an associated probability P(C|p)--where C is the 
class of patch types (IDC and Non_IDC), which is obtained from the classification carried out in the previous 
process. This probability is exploited as a prior probability in MRF process. Then, an inference is caried out 
using graph cut techniques to obtain the final classification, as the refinement procedure. Given a set of 
image patches W, we can define a lattice graph and generated it as shown in Figures 3(a) and (b). The edge 
eij defines the adjacency of pair of the patches i and j in the graph. To obtain a final classification, an energy 
function is defined as: 


E(G) = Es + Ea.) (4) 
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where E, denotes a function that examines the smoothness of the connected nodes. This energy term usually 
preserves the discontinuity at the boundary between classes. Eù is defined as (5): 


E, = Le jen iz Vpop), (5) 


where V is a smoothing function preserving the relation of the connected patches. V(p;, p;) is calculated 
based on radial-based function, which is defined as (6): 


{hex} 
V(p;, pi) =e (6) 


where o signifies the offset of the relation between the image patches. In this work, the data term is 
designated by using the posterior probability of the classification performed in the previous step. 


Ea = Les D(p) (7) 
The energy function shown in (4) will be minimized subjecting to the class label (C) to obtain the 
optimal results, as the final classification decisions. Therefore, the data term is calculated based on a function 


defined as (8). 


D(s) = —log(c|s) (8) 


To obtain the final classification decision, the optimization is carried out using the graphcut technique [22]. 


x 
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Figure 3. Example of the relation of image patches in the image slides; (a) the spatial relationships of a patch 
and its neighbors bases on a distance metric and (b) the 8-neigbor of the patch 


1 
16541: 


(a) (b) 


4. EXPERIMENTAL EVALUATIONS 
4.1. Experimental setup 

The image patches (from the whole slide) are prepared. Background patches are removed from the 
dataset using a simple thresholding method on the normalized image patches [0-1]. The threshold (T) is set to 
0.2 where the original background is 0. All the image patches are divided into 2 set, i.e., IDC and non-IDC. 
The dataset is imbalanced when negative patches are greater than positive patches. The data is separated into 
2 sets, 70% of the dataset is the training and validation set, and 30% is folded as the test data. 


4.2. Experimental result and discussion 

The experiments are conducted to evaluate that performance of the proposed method. The 
handcrafted features are generated on the dataset (explained in section 3.) before the classification is carried 
out by a random forest. The experiment varies the size of a number of trees in the random forest to observe 
the sensitivity of the classifier. The classification results are demonstrated in Table 1. 
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Table 1. The results of the classification obtained from the handcrafted-features (textural features and HOG) 
and a random forest with a various number of trees 


No. of Trees = 50 No. of Trees = 75 No. of Trees = 100 No. of Trees = 125 
precision recall precision recall precision recall precision recall 

GLCM 0.75 0.72 0.78 0.71 0.76 0.73 0.77 0.75 
HOG 0.69 0.70 0.75 0.75 0.71 0.74 0.74 0.72 


From Table 1, the precision and recall value (average over two classes) obtained from different size of 
the Random Forest are marginally the same. GLCM produces better results, in overall, than HOG. No. Using 
GLCM, No. of Trees (NoT)=75 outcomes the best precision (0.78) yet produce low recall (0.71) comparing to 
NoT=125 (which is 0.75). Therefore NoT=125 will be used for the comparison the later experiments. The CNN 
(explained in section 3) is implemented. The defaults parameter of VGG16 is employed to construct the 
network. The training and testing are performed on the prepared dataset and the resulted is shown in Table 2. 

Table 2 show the results collected from using VGG16 (in the classification process). The results 
demonstrate the deep learning implemented in this work outperforms to the handcrafted features shown in 
Table 1. The precision and recall values are increased up 4-6%. To evaluate the performance of the proposed 
method, the last experiment is run on the same dataset. The post-processing (explained in section 3) is 
implemented. The classification in the single patch-level classification is deployed in the optimization 
process using the graph-cut technique. The results are shown in Table 3. The results in Table 3. show that 
pre-processing technique that use the relationships between the patches can improve the performance of the 
classification. 


Table 2. The results of the classification obtained VGG16 


Positive Negative Average 
Precision Recall Precision Recall Precision Recall 
Validate Set 0.89 0.87 0.82 0.82 0.85 0.84 
Test Set 0.84 0.82 0.79 0.81 0.81 0.81 


Table 3. The results of the classification obtained from the post-processing process by integrating the spatial 
relationships of the image patches 


Single Classification Post-Processing 
Precision Recall Precision Recall 
VGG16 0.81 0.81 0.85 0.82 
GLCM+RF 0.77 0.75 0.81 0.82 
HOG+RF 0.75 0.75 0.79 0.78 


5. DISCUSSION 

Identifying the region of cancerous arears can be achieved by implementing a classification 
technique. This work tries to solve a binary-class classification by dividing the whole slide into two areas, 
IDC, and Non-IDC. Therefore, the image is separated into small non-overlapped patches. From visual 
inspection, there some subtle different of appearance of the patches. Therefore, a textural feature is used. 
GLCM can deliberately present the overall appearance of the patches and result fair outcomes. The precision 
and recall of the two is around 75-77%. Considering the results, GLCM and HOG produce low recall in 
negative samples (70%). This may not be applicable in implementing the technique in actual screening 
system, where we try to highlight the cancerous region for the pathologists. In addition to the handcrafted 
features, Table 2 demonstrates the results obtained from the deep learning method (using VGG16). From the 
observation, VGG16 produces marginally good results. It obtains 82% of recall on negative samples and 
results good outcome for positive samples. The technique generates the abstraction of the features (deep 
features) from the images, which increases the discrimination degree in this binary-class problem. 

Table 3 demonstrates the importance of applying the spatial relationships of the patches (location 
and the output from the classification) in the whole slide images. Miss-classification can be found throughout 
the patch-level classification. The post-processing is implemented to correct the miss-classified patches from 
the classification. The results indicate that around 4% of miss-classified patches are corrected. These image 
patches are assigned a class label based on the result of the patch-level classification. The classification result 
of the two class can be very subtle for this binary-class problem, for example 0.49 for IDC and 0.51 for Non- 
IDC. Therefore, apply the spatial location of the patches tends to improve the performance, as a sing patch 
label is likely to be identical to the patches around it. 
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6. CONCLUSION 

Invasive ductal carcinoma (IDC) is the most common subtype of all breast cancers. In general, 
pathologists identify IDC in the tissue images and evaluate the severity of the tissues. In addition, identifying 
IDC in the slides is a challenging task, as the appearance of IDC can be subtle to other regions/areas. This 
work proposes an automated technique for identifying IDC in the tissue images using a classification 
technique. A tissue image divided into small non-overlapped patches (or image windows). Then, the task is 
to classify the image patches into different classes, i.e., (i) IDC and (ii) non-IDC. This work employs a two- 
stage classification-based to classify the patches, as to identify IDC regions in the tissue. In the first stage 
(patch-level), image patch classification is carried out using a conventional handcrafted feature (based around 
on texture descriptors) and deep-learning technique are explored. The second stage undergoes a refinement 
process, which considers the spatial relationships between the neighboring patches. This stage aims to amend 
some of miss-classified patches to correct classes. Markov random field (MRF) is implemented in this stage, 
to take in to account the relationships of the patches nearby. The experiments are conducted on the collected 
data of the image patches. The experimental results show the post-processing process is superior to the patch- 
level classification using the handcrafted-based technique and deep learning. 
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